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ABSTBACT 

The development of A Simulation in Educational 
Evaluation (SIHEDEVAL) , a simulation designed to increase the 
evaluation capability of school personnel is described. The 
Discrepancy ^Ivaluation Model served as the simulation model and the 
framework for field test procedure. Instruments were constructed and 
validated . for use in the field test with graduate education students, 
fiesults indicate that SIMEDEVAL inc^reased students' knowledge of 
evaluation methodology and willingness to engage in evaluation, and 
demonstrated their ability to apply evaluation techniques in a 
structured setting. It is concluded that SIMEDEVAL warrants further 
deyelopment, and that simulations* can be subjected to evaluation, 
procedures. (Author/EC) 
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Introduction 

This paper provides an overvie^of the development of a simulation designed 
to increase the internal evaluation capability of elementary and secondary 
education personnel at a relatively low cost in terms of both time and money. 
Developpiental stages include construction of the. simulation, development of 
an evaluation design for the field test of the simulation, instrument construction, 
and the field test itself. . 

Construction of the Simulation 

A Simulation in Educational Eval uation (SIMEDEVAL) was constructed according 
to guidelines established by Twelker (1969) and McGuire et. al. (1972). These 
guidelines may be summarized as a series of steps; (1) define the instructional 
problem, (2) describe the operational educational system, (3) relate the operational 
system to the problem, (4) .specify objectives in behavioral term"s, (5) generate 
criterion methods, (6) determine the appropriateness of simulation as a method, 
(7). determine the type of simulation required, (8) determine specifications 
for simulation, (9) develop the simulation .prototype, and (10) field test the 
simulation prototype. While each step in the construction of SIMEDEVAL will 
not be detailed, some attention. is given to the selection of simulation as the 
instructional method and the determination of the type of simulation required. 

From review of the literature on simulations, four conclusions can be 
drawn: 

\- . ■ 

-rSimulations are interesting to participants (Cruick shank, 1972; Abt, . 
1970; Taylor & Walford, 1972; Cherryholmes , 1966). 

--Simulations can influence participant attitudes andxyalues in a given 
direction t'Heinkel, 1970; Fuller, 1973; Stadsklev, 19t3; Coleman, et. al., 
1973). \ 
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—Simulations are neither more or less effective than traditional methods 
in teaching factual information or conceptual knowledge (Cherryholmes , 
1966; Heinkel, 1970; Fuller, 1973; Stadsklev, 1973; Coleman, et. _al . , 
1973). - 

—Simulations are most effective when they are used for training purposes, 
and as a result of the simulation, the participant is expected to perform 
a specific task in a specific manner (Elder, 1973; Colemanv et. al . , 1973). 

Thus, simulation is most appropriate as a method of i nstructi on .when objectives 

^ ... 

are to influencu participant values and attitudes in a given direction, and 
also to train participants to perform a specific task in a specific manner. 
Given the author's objectives of producing a training vehicle which would 
serve to increase participant's knowledge of evaluation methodofogy, to broaden 
their concepts of evaluation, to increase willingness to engage in formal 
evaluation, and to provide application of specific evaluation techniques, . 
simulation was determined to be an appropriate method of instruction. 

The selected format of the simulation can be classified as media-ascendent 
according to a system developed by Twelker (1969). v,As a media-ascendent simulation, 
SIMEDEVAL lacks the common characteristics of interpersonal -ascendent simulations 
such as emphasis on ..deci sion-makjng, 'competition, and player interaction. 
Instead as a niedia-ascendent simulation, SIMEDEVAL is characterized by the 
instructional burden carried by media, in this case by programmed instruction. 
Major advantages of this format (Twelker, 1969) are that the simu^lation can 
be used where learners possess low entry skills and limited response repertoires 
that would prevent an interpersonal -ascendent simulation from ^el 1 , 

minimal participation is required by an instructor, learning objt as are 
generally congruent with staiidard course objectives, and a relatively large 
amount of control is possible. ^Given the intended audience and the objectives 
of the simulation as well as the- desi re -for a step-by-step incremental training 
progression, the media-ascendent simulation was selected as most appropriate. ; 
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A Simulation in Educ ational Evaluation Is based on the Discrepancy Evaluation 
Evaluation Model developed by Provus (1971). The Discrepancy Evalua^tion Model, 
or the "DEM", includes (1) provision for continuous feedback, (2) procedures • 
for program planning, (3) procedures for examining program implementation, 
(4) procedures for monitoring program processes, and (5) procedures for program 
assessment. In the simulation the concept of the team evaluation presented ^ 
in the DF.M is modified so that it describes staff members i,nternal to an educational 
program who are involved in its evaluation. The simulation also introduces 
the concept of the participant evaluator; a staff member who is involved in 
the daily operation of the program and also has responsibility for its evaluation. 

Within the simulation the participant plays the role of a public school 
staff member who needs to develop evaluation skills in order to take responsibility 
for the evaluation of a Title III program for the gifted. He is assisted 
in his endeavors by an evaluation consultant who, utilizing a learning-by-doing 
method, leads the staff member through major steps in the discrepancy evaluation 
process. The staff member learns to produce a program design (a method of 
program description) and an evaluation plan for an_.educational program. 

Participants in SIMEDEVAL work through 10 tasks in sequence beginning 
with a general overview of discrepancy evaluation and exposure to basic terms 
and concepts and concluding with application of the techniques of program design 
and evaluation planning. Each of the tasks uses a paper. and pencil format 
and is divided into several segments: a scenario for the task, instructions 
for the task, materials, response forms, standard response forms against which " 
the participant can compare his/her performance, and a feedback form. A total 
cf ten hours is estimated for simulation completion. 

The simulation is designed. to serve as a basis of a course or workshop 
with students working individually or in groups with the supervision of an 



instructor or trainer. While SIMEDEVAU has potential use for sol f-instruction, 
its recorninended use at this time is with supervision. 

Evaluation Design 

/* 

"fi addition to serving as the fiamework for the content of the simulation, / 
the Discrepancy Evaluation Model (DEM) also served as the framework for the 
evaluation of SIMEDEVAL in the field test situation. A Program Design, based 
on the DEM, was constructed to guide the a'dmi ni strati on of the simul ation, 
and an Evaluation Plan, based on the same model, was constructed to guide data 
collection and analysis. ^ 

The' program design for SIMEDEVAL consisted of two levels. Le' el I presented 
an overview of the expected inputs, processes, and outputs; that is, it described 

o 

the resources or preconditions necessary for field test of the simulation, • « 
the major' activities or tasks within tne simulation, and the expected terminal 
or final outcomes for the simulation. Level II detailed the inputs processes, 
and enabling or interim objectives for each task of the simulation. The program 
design not only served as a procedural guide for administration of the stmuTation, 
but also served as the basis for evaluation by providing the standard against 
which actual performance tbuld be compared. The evaluation plan for SIMEDEVAL 
delineated the areas of concern addressed by the field test, the specific 
evaluation questions asked under each area of concern, 'and for each evaluation 
question, the standard against which the performance was to be compared, the 
source of the information, the instrument to be used to collect the information, 
proposed data analysis , and date(s) .for instrument administration. 

■ In brief, the evaluation design for SIMEDEVAL was intended to provide 
Information" as to the initial value or face validity of the simulation, the , 
^fec-tiveness of the simulation within the field test situation, and needed 



areas of r.vUion «ithin the simulation.' To detemlne the effectiveness of 

■ the sinulation within the field test, four hypotheses (also terminal objectives 

of the' simulation) were tested: 

-There will be a significant difference in participants' scores on a. 
pre- and post-test measure of evaluation concepts. 

procedures, 
determined criteria. 

instrumentation 

Instruments to, be developed for the f1#d test of SIMEDEVAL Included ^ 
feedback for^s.' t1» sheets, biographic Information forms, as well as Instruments 
for determination of the effectiveness of the simulation. An Evaluation To=- 

■ was constructed tg^determlne each sl^Tatlon participant's knowledge of evaluation 
methodology, and an Evaluation Attitude questionnaire was constructed to determine 
both participants' concept of evaluation and willingness to 'engage in formal 
evaluation. An Evaluation Rating Scale was al.o developed to determine initial 
value of the. si^lation. Criteria for determining the adequacy of each participant's, 

■ program design and evaluation plan were incldded within the Simula .n. although 
additional .scoring guidelines for use of outside raters were developed. 
• TheEvaluationTest. used as the measdre of knowledge of evaluation 

n^thodology, consisted of 16 multiple choice items. The items were derived 
-» ■ from analysis of texts by VTorthen and Sanders. (1973) and the PDK National 
Study Co-ittee on Evaluation (1971),. and were based on the idantifi cation 
of mador steps necessary In conducting formative and summative educational 

o ■ ■ , 7 ■ 
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evaluation. Information concerning the reliability and validity of the instrument ' 

was Obtained from 28 gra<?u te students in education at the University of Virginia. ■ j 

A reliability coefficient of .79 wBs obtained for the Evaluation Test. In j 

addition. 8 of the test's . :tems were found to discriminate at a .05 level 

of. significance between graduate students with and without a major in educational 

evaluation. 

The Evaluation Attitude questionnaire, used as the measure of concept 
of evaluation and indicator of willing^ness to engage in formal evaluation, . • 

consisted of >^o scales. The first, or "concept" scale consisted of 16 items 
derived from^fs (1972. 1973) work with public school teachers. Wolf identified 
%• variables .X discriminate between indivi dual s^Tol ding a broad or narrow 
' concept of evaluation. For example, a broad concept of evaltiation is held 
by teachers who consider the judgements and opinions of students important- in 
evaluation.of programs, As adapted for theEvaluation Attitude Questionnaire, 
participants were to indicate their agreement/disagreement on a 5-poinf scale 
with the importance of student opinion in program evaluation. While a reliability 
coefficient" of '.82 was obtained for the concept scale, the scale failed to . 
discriminate between evaluation and non-evaluation graduate students. 

The second, or "willingness" scale consisted of 2 items. Using a ; 
5-point rating system, participants were to indicate their willingness to 
engage in'formal program evaluation. " A reliability coefficient of .82 was 

Obtained. In-addilion. these items did discriminate between evaluation and . 

non-revaluation graduate students. . 

As a measure of the initial value of the sirajlation, , a Simlation 
Rating Scale was developed. The. seal e was developed fron, an analysis of the ^ 
literature which ■ resulted in the identification of 10 criteria for determining 
the initial value of the simulation: instructional value, validity: comprehensiveness 



verisimilitude, symmetry, synchronization, manageabil ity, ease of administration, 
immediacy of feedback, and presentation' of learning objectives. 'These criteria 
were used as the basis for the rating scale. 

/ Field-Test Of the Simulation ' . _ .. 

Methods ' . 

The field test of the simijlation took place at the School of Education, 
San Jose State' Col 1 ege/ SIMEDEVAL was offered tb graduate level students as . • 
a Special Studies one credit, 16 hour course. The course was taught by the - 
author from 1:00 to 5:00 pm d&ily, June 24 through June 27, 1975. At the 
beginning, of the /irst session, participants were asked' to complete the pre- 
measures (Biographic Data Questionnaire, Evaluation Attitude Ques+^'onnaire, 
and the Evaluation Test). Evaluation Attitude Questionnaires were precoded 
to allow for anonymity of response. After an introduction to the simulation 
by the instructor, participants were allowed to work *i ndi vi dual ly-, proceeding 
at different rates. Neither formal teaching nor discussion sessions were 
offered by the instructor, although individual of group requests for clarification 
o,f a task v/ere met and recorded. Nofie of the simulation material was used 
for formative purposes during the field test, that is, a student who did not 
meet the standard on Task One was not given remediation before continuing 
with' Task Two. Upon completion of the- simulation., participants were asked 
to complete post-measures (Simulation Rating Scale, the precoded Evaluation ■- 
Attitude Questionnaire and the Evaluation Test). . 

During the week^of July 21, ip^^ evaulation faculty, staff, and advanced 
graduate students were asked to examine SIMEDEVAL briefly and rate its initial 
value. ' - 
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■/ During'the week of Ju.ly^28, 1975, . two dpctoral lev,e-l graduate'- students 
in evaluation and the author 'independently scbred the program designs and 
evaluation, plans produced by the simulation participants in Tasks'9 arid IG- 
of the simulation. Each rater was gi ven a set of scoring gui deli nes." Each 
Participant's final score was the average of the 3 ratings for each task." 

Results " 

.' The 10 students participating in. the field test of SIMEOE'VAL were typically 
female, between 41 and 50 years of age, and elementary school teachers with 
many years of teaching experience and-graduate level coursework. Ovei" half 
of the students had been part of writing teams whose task was to develop Early. 
Childhood Education or Individually Guided Education program or program designs. 
Three questions were asked of the field test of SIMEDEVAL.- (A) What - / 
' changes arejndicat^d either in the simulation or its -administration to improve* 
its effectiveness?, (B) How effective is the simulation within the f iel d-tfesting 
situation?, and (C) What is the initial value of the simulation? In response 
to' the first questiop, the field-testing of the simulation indicated several 
areas of- needed revision. These included r^i^on of several lengthy- tasks 
•toi)ring the simulatin within the 10 hour desired completion time, clarification 
of instructions and response formats for several of the "tasks, and modi fi&a Jt ions 
in the author's standards for several of the tasks. ^ 

Determination of the effecti veness^of the simulation required examination 
of the .4 hypotheses stated previously. First, as a measure cf -the ef fef ti venuss 
of» the simulation, the hypothesis was stated that students can produce both 
a program design and an evaluation plan for an' educational program which meets 
■a predetermined standard. Two doctoral level graduate ^valuation students 
and the aufhbr independently scored 'the simulation participant's products 
using pre-established scoring gu'^idel ines and xviteria. Participants were ' 

. ■ ' ■ ' ' ■ ' ■ 10 ■ ' ■ '■ 
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assigned the average of the 3 rf->tings for each tasl^ . All 10 participants 
met t1ie standards established for the'- products. • • 

A/the sedbnd indicator of t"he,^f fecti veness of SIMEDEVAL, the hypothesis 
was. staged' that there would be *a significaj;<t di f ference. i n participant's scores 
6n a pfe- and post-test measure of knowledge of evaluation methodology. Only 
those items of the Evaluation Tesf which were Ifound, tp dtseriminate between 
graduate stiideijts.!,in evalaation a.nd^non-evalu^tion graduate stijdents we^e - - - 
r use^d in the a'aalysis. Results indicated a <lif ference^i n . the partici pants' ' ^" . 
•scores on the pre- and post- measure of knowledge-pf evarluation/methodp-logi' ^ 



significant at the ^05 level... ^ v 

As a third iad^icator of the effectiveness of -the simulation^, the hypWiesis 
was stated that participants* scores on a pre- and post-test mea§u.re of concept 
of evalMationS/ould differ. While at the .05 level of signific^nce^^uch 
/a difference was found, the lack of demonstrated validity of the Evaluation , 
Concept Scale precluded acceptance of the hypothesis that simulation participlints 
broaden concept^-of evaluatt^on. ' ^ 

Finally as the fourth indicator of. the effectiveness of SIMEDEVAL, the 
, hypothesis was stated that simulation participants' scores on a pr*e- and post- ^ ^ 
test measure of willingness to engage in formal evaluation would^di ffer si..gni fi cantly , 
Results indicated that participants in the simulation did increa,$e in l/il1ingness 
to engage in formal program evaluation at a .05 level of significance. 

-The..jym^d^ asked in the field test of SIMEDEVAL concerned the^ 

Initial value^'^or face validity of the simulation. Simulation participants 
and evaluation graduate students, staff, and faculty gave SIMEDEVAL high ratings- 
on all 10 criteria presented in the Evaluation Rating Scale. In addition, 
many of the simulation participants commented that it had been an^interesting, 
valuable, and challenging experience. : 



Conclusions 



■ The development of SltCDEVAL offers three major contributions. First, ^ 
the study indicates that an evaluation model such as-tjie DEM can be used to 
evaluate simulations. AppVication of an evaluation model can provide information 
to improve the simulation itself as well 3S to assess its effectiveness, in 
a given situation. Second, the work has produced-an .evaluation knowledge 
test and a scale which measures will ingness to engage in formal evaluation 
which have demonstrated reliability and validity. Beyond their application ■ .. 
to SIMEDEVAL, the instruments may be useful in examining the effectiveness 

\ " . ■ ■ . . 

of other evaluation training procedures. Third, SIMEDEVAL looks like^a promising 
•vehicle, for increasing the internal evaluation capabil ity of elementary and 
secondary school personnel. While investigation has. been limited to one group 
of students, the simulation was effective in increasing 'participants' knowledge 
of evaluation methodology, willingness to engage in formal ^Valuation, and . 
ability to produce program designs and evaluation plans whiph encompass both 
formative and summative evaluation procedures. .In addition,*^ SIMEDEVAL hac 
initial value in terms of criteria established for good simulations as rated* 

f ^ . 

by both' simulation participants and evaluatipn specialists. In sum, it can 

be concluded' that A Simulation in Educational ' Eval uati on ^mer.itg. further development 

and testing. ^ . \ ' ^ 
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